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DIRECTOR’S NOTE 



After the Western Regional Center published its six-part series of articles on systemic evaluation of 
tobacco, alcohol, and other drug (TAOD) programs in our quarterly newsletter, the Western Center 
News, we began receiving numerous requests for reprints. This publication was produced in 
response to those requests. The reprints will become part of the center’s final product, A 
Comprehensive Approach to Building Prevention Programs: A Guide For Trainers. We are 
confident that the systemic evaluation series will be so useful and so packed with practical advice 
on how to approach TAOD program evaluation that it will become a permanent part of every 
preventionist’s reference collection. Taken as a whole, the articles in this collection represent 
cutting-edge information on systemic evaluation approaches that will become a model for use in 
the program consolidation efforts now in progress across the region and throughout the country. 



Many thanks to authors Dean Arrasmith, Kim Yap, Jerry Adams, Steve Nelson, Joan Shaughnessy, 
Roy Kmger, Patrick Weasel Head, and Changhua Wang for their fine work. Many thanks to Lee 
Caudell for her excellent editing and to Denise Crabtree for her contribution to formatting and 
proofreading the articles. 

Carlos Sundermann, Director 

Western Regional Center for Drug-Free Schools and Communities 
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INTRODUCTION 



Evaluations that provide a static, discrete, isolated assessment of the effectiveness of a program or 
project are generally neither meaningful nor useful to policymakers or practitioners. The 
shortcomings of current practices can be remedied by switching to a systemic approach that looks 
at projects and programs within their larger context. Such an approach viev.-s programs and the 
environment in which they exist as fluid, dynamic, and interactive systems affected by multiple 
influences at many levels. Changes in the pattern of interrelationships among such influences as 
peers, family, school, and community in turn affect project results. 

This series of papers desaibes what we consider to be essential features of systemic evaluation. 
Developed on the basis of a review of the relevant literature, discussion among evaluators at the 
Northwest Regional Educational Laboratory, and our experience with program evaluation, the 
papers are intended to stimulate and facilitate future research and development work on systemic 
evaluation and the use of indicators of progress in complex school and community improvement 
situations. We wish to share our emerging perspectives with interested policymakers and 
evaluators to begin a dialogue on developing a systemic approach to evaluating interventions in 
education and other social service areas. 

Systemic evaluation, in our view, is particularly applicable to the evaluation of AOD programs 
where multiple sources of influence (e.g., peers, famUy, school, and community) impinge on the 
success of prevention and intervention efforts. 

These articles were originally published as a series beginning in September 1994 and running 
through September 1995 in the Western Center News, the quarterly newsletter of the Western 
Regional Center for Drug-Free Schools and Communities. 



Kim Yap 
Unit Manager 

Evaluation and Assessment Program 
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Article 1, September 1995 



Programs Need to Go Beyond 
'Comprehensive' 

By R. JERRY ADAMS and KIM YAP 

Northwest Regional Educational Laboratory 

When you say your prevention program is “comprehensive,” whal do you mean? Do you 
mean the program contains multiple components? Do you mean the program involves parents, 
community members, and peer groups? Perhaps more importantly, do you fully understand how 
your comprehensive program works — how its components relate to each other and how they 
interact with the larger system of w'hich the program is a part? 

Without knowing hov/ your program works and how it interacts with the larger environment, 
it is difficult, if not impossible, to know how to strengthen it and how to make it interact more 
positively with the larger environment or system to produce the desired outcomes. 

Many preventionists argue that alcohol and other drug (AOD) programs are most effective if 
they are comprehensive. That may mean involving school, community, and/or family. It may mean 
including a number of strategies and activities. Such broad-based approaches can go a long way 
toward effeaive prevention. We suggest, however, that there is yet another level or degree of 
programming that embraces all players holding a stake in AOD-free youth and all the 
environments they represent. Such an all-encompassing approach can be termed “systemic.” 

We want to turn this discussion to the question, Whal does it take to make the difference? 
Specifically, what are all of the pieces that should be in place and how should the pieces fit 
together? How should the school work with families? How should the community v;ork with 
schools? What kinds of activities should be in place for good peer support? 

A systemic approach includes many strategies and is, by defmition, comprehensive. A 
comprehensive approach, however, may not necessarily be systemic. For example, a school may 
have AOD instruction embedded in several subject areas, may have a D.A.K.E. program, and may 
sponsor some AOD-free recreational or sporting events. The school’s administrators may say that 
their approach is comprehensive. Another school may also involve parents in the planning of 
some AOD-free recreational activities and consider itself even more comprehensive. A third school 
may also involve community leaders in a mentoring program and be more comprehensive still. 

A fully comprehensive program would include all of the essential strategies, aaivities, and 
environments— school, community, family, students, and peers. However, including all of the 
strategies, activities, and environments is not enough to make a program systemic. In order to be 
systemic, the relationships among the components also need to be developed. In short, a 
“systemic approach” covers all of the essential strategies and activities in all of the essential 
environments — and the relationships among them. It ensures that the various strategies work in a 
coherent way, pushing in the same direction to achieve the desired outcomes. 

A systemic approach suggests the need for a chart so the components and relationships can be 
visualized (see sample chart on page 3)- Such a chart allows community, school, family, and 
student representatives to discuss and clarify their beliefs about what the essential strategies are. In 
addition, the chart allows them to check their strategies against findings from research on whal 
has worked. 



Next, the participants in program planning can write in the activities that are currently in 
place. Do those activities seem to be sufficient to fully implement each of the essential strategies? 
If not, what activities, in which environments, must be added? 

Examples of strategies include promoting positive expectations, ensuring care and support, 
and generating meaningful involvement. These examples come from the literature on promoting 
resiliency in children. For each strategy, a series of activities or programs may need to be in place, 
depending on the community. Some strategies may need several programs, and some may need 
few. Some strategies may need to be implemented in all environments and some may not. (The 
chart below provides some examples of activities that may be used to implement a strategy.) 

Once the components of the system have been portrayed, then planning and evaluation 
activities can be developed to monitor progress. For instance, each cell of activity may be 
evaluated in terms of its impact. The chart also promotes discussion of the relationships across 
strategies, activities, environments, and stakeholders. The chart becomes a tool for simcturing a 
systemic planning and evaluation process that includes the following steps: 

1. Clarify the problem or needs 

2. Access and synthesize relevant knowledge 

3. Develop and apply strategies to make systemic improvement 

4. Design a systemic evaluation process to monitor progress and assess impact 

5. Implement the systemic evaluation process (including data collection, analysis, and 
reporting) 

6. Use evaluation information to strengthen strategies for systemic improvement 

A systemic improvement has several essential characteristics. Specifically, an improvement is 
s 'Stemic when it (a) is accepted within the components of the existing system, (b) spreads 
through the entire system, (c) has power to make visible, desired differences — both in tem^ of 
process and outcomes, (d) strengthens the linkages between and among components of the 
existing system, and (e) becomes part of and improves the culture of the system. 

A systemic approach to your comprehensive program is fluid and dynamic, witli many levels 
of interactive and causal influences. Its impact can best be understood and explained in terms of 
changes in the pattern of interrelationships among the relevant variables at multiple levels 
(individual students, peers, family, school, and community). 

The chart may be the first step in a six-step process of systemic planning and evaluation. Each 
step will be discussed in a separate installment of this series. 
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Article 2, December 1994 



Systemic Planning of Goals for 
Desired Outcomes 

By STEVE NELSON 

Northwest Regional Educational Laboratory 

As a beginning point for systemic evaluation, we can take a more systemic perspective in 
planning the goals of our improvement efforts. While educational planning usually begins with 
considerations of needs assessment and problem clarification, a systemic view will broaden this 
phase to look at the “what” and “why” questions about the desirable scope of the goals we wish 
to achieve. Goals reflect not just the minimization of identified needs, but also the attainment of 
desired outcomes. In other words, our desired outcomes must become more inclusive and 
divergent, rather than behavioristic and convergent. 

We must recognize from the outset that individuals and organizations are complex, so our 
understanding of the nature of our goals and solutions must be carefully integrated. As Healthy 
People 2000 points out: “I'he most effective community-based health promotion programs 
recognize the interrelationships between behavior and environment, and include multiple 
interventions directed at multiple levels (e.g., individuals, small groups/fainilies, organizations, 
community). This is true regardless of the target population or the issues being addressed.” 

So, while it may not sound much like evaluation, we need to spend time analyzing the 
systemic nature of our goals. Do our goals adequately reflect system outcomes? 

The systemic planning of program goals has at least two phases: (1) defining the nature and 
etiology of the goal to be addressed and (2) establishing a rationale that defines the contextual 
relationship of the goal to our system. Put more simply, what goals do we want to achieve, how 
did these goals come to be, and why are these goals associated with the funaions of our 
organizatioa^ 

Defining the Nature and Etiology of the Goal 

Back when experimental psychologists ruled the earth and behavioral objertives could be 
precisely deFmed and measured, educators were encouraged to plan and evaluate in a needs- 
driven environment. Specific, individual needs were assessed and remediated. A highly 
sophisticated, deficit-oriented system was developed for eliminating undesirable behaviors. 
Unfortunately, this orientation does not work particularly well in a health promotion setting, 
where an array of desirable knowledge, skills, attitudes, and behaviors is expected to be achieved 
and applied. The traditional needs-driven approaches do not lend themselves to universally 
inclusive goals for well-being because of their focus on deficit reduction. This is not to say that 
some elements of our goals should not have a risk-reduaion focus. Rather, desirable outcomes 
should also be defined in the goal. For example, "Health is more than the absence of disease.” 

So, practically speaking, how do we define the nature and origins of our program goal? There 
are four probable origins from which program goals would most likely derive; 

1 . Mandates from regulatory agencies 

2. Health needs of students documented in the local area 

3. Values expressed by the local community 
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4. Standards advocated by professional organizations 

Our program’s goals most likely originated from one or more of these sources; What outcomes 
are required by regulation? What vital statistics or incidence and prevalence data document student 
needs? What health-related values are expressed by community stakeholders? What student 
perfomiance standards are embraced by professional associations? Grounding our goals within the 
context of these sources of information will help us to think about the systemic relationship of our 
program to that of other systems. 

But how can we tell if the goal is systemic in nature? Besides being integrated (grounded in 
the various systems noted above) the goal needs to be inclusive, encompassing all children and 
incorporating the full range of outcomes from promotion to prevention to intervention to 
reintegration. This orientation requires us to think about the scope and severity of the need to be 
addressed. How pervasive is the need? What is the consequence of not achieving the desired 
outcomes? What would the desired state ultimately look like, not just as the absence of the need, 
but the absolute presence of the desired trait or condition? This step is a process of reconciliation 
in which the various facets of the goal are considered and brought into alignment. 

Establishing a Contextual Rationale 

This step helps to clarify .he conneaions between the goal to be achieved and the functions 
of our organization. What legitimate roles might our organization play in addressing the desired 
goals? What reasonable and desirable steps could be undertaken by the organization to achieve 
the intended outcomes? "Think about wfiat contextual conditions can be affected by the program to 
encourage the intended outcomes. Having a rationale as to why a program is addressing this goal 
will help to develop more holistic, multiple paths and strategies for improvement. 

The goals of our program are more likely to be systemic in nature if they are: 

Sustainable— -Would the organization continue to hold this goal in the absence of any special 
funding or programs? 

Potent— Is this goal understood, valued, and embraced by stakeholders in our community? 

Infusive — Does this goal build upon the existing mission, resources, and infrastructure of our 
organization? 

Pervasive — Is the goal multifaceted and inclusive in its desired outcomes? 

Coherent — Does the goal create and encourage congruence among the levels and dimensions 
of our system? 

In this first phase of systemic evaluation, we are determining the extent to which the goals of 
the program are systemic in nature. In the next phase, we will determine the degree to which the 
program’s planned intervention strategies also embody systemic principles. Thus, in the early 
stages of systemic evaluation, the purpose is to determine if the improvement effort itself is 
systemic. 

References 

Hagans, R., Crohn, L., Walkush, L. & Nelson, S. (1992). The states’ role in effecting systemic 
change: a Northwest depiction. Portland, OR; Northwest Regional Educational Laboratory 

Nelson, S.R. (1986). How healthy is your school? Guidelines for evaluating school health 
promotion. New York; NCHE Press 

Nelson, S.R. (1987). Clarifying and defining the needs of schools: a process approach for school 
improvement organizations. Portland, OR; Northwest Regional Educational Laboratory 
U.S. Public Health Service. (1991). Healthy People 20G0, national health promotion and disease 
prevention ob/ectiws. Washington, D.C.; author 
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A Framework for Cooductin 
Systemic Evaluation 

ByDEANARRASMITH 

Northwest Regional Educational Laboratory 

A useful tool for helping develop program logic is the “logic model.” This tool is designed to 
help stakeholders in a system such as a community, a school, or an organization to think together 
about how their project can effea change in the health of the system. In a community drug 
improvement project, the logic model can be useful in eliciting a shared understanding of the 
current condition of the community and in planning activities that are likely to result in desired 
outcomes and impact on the community’s health. Throughout the model, rational links based on 
shared craft knowledge and empirical research are established, helping to guide the development 
and implementation of projea activities. 

The development of the model provides an opportunity for establishing a shared understanding 
of the project logic. Thus, logic here means the rational, empirical, and intuitive relationships 
established among system context, inputs, processes, and outputs that lend validity to the possible 
effectiveness of the interventions. 

The basic logic model consists of four linked components of the project: (1) the context or 
condition of the system, (2) the interventions or activities to be conducted, (3) the expected 
outcomes related to the interventions, and (4) the impact of intervention on the system. An 
example logic model is presented in the figure on page 10. 

The first component of the logic model is the depiction of the current condition of system 
health. For example, in a community improvement project, the health of the system may be 
indicated by information about suppon services, community participation, crime statistics, student 
academic success, juvenile offenses, community health programs and health problems, economic 
development data, housing availability, and use of social assistance. The specific indicators of 
community health should be identified by key stakeholders who each have a perspective on the 
health of the community and who will, through the activity of developing community health 
indicators, develop a shared understanding community health. 

A list of possible indicators for a profile of the community’s health is shown in the first column 
of the fipure below. This list was developed by stakeholders in a community drug- and alcohol- 
prevention project. These indicators provide a shared understanding of the strengths and needs of 
the community related to drug and alcohol use, especially among youth. The community profile 
allows stakeholders to share their undersLtndmg of the community’s health with others in the 
community and to facilitate the development ol project activities that build on identified strengths 
and address identified needs of the community. 

The second component of the logic model is the specification of the activities to be 
implemented by the project. These activities need to build on the understanding of the strengths 
and needs of the system and target the desired changes in the health of the system. Because of the 
requirements that activities relate to community strengths and needs and be directed toward 
desired changes, the process of identifying specific activities often is an iterative process (a 
repeating, refinement process) rather titan a linear process, arising from consideration of the health 
of the system and consideration of the desired changes in the health of the system. This iterative 
process will become clearer as we discuss the identification of outcomes, impaa, and project 
logic. 

Our experience in working with several community and school projects is that stakeholders 
often have many activities they want to implement, and the development process often is one of 
focusing on a set of activities that are most likely to be efficient and effective in promoting the 
desired changes in the health of the system. This is where program logic can be most useful in 
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developing shared criteria for selecting activities among the many that are likely to be proposed 
for a project. In fact, we have worked with some communities that now use their established logic 
models to screen requests for new activities. If the activity fits into the program logic and there are 
sufficient resources for the aaivity, the project embraces the activity. However, if the activity does 
not fit the program logic, then the activity is rejected or, if necessary, the program logic is 
reviewed and modified. In one example, a project continued to receive requests for multicultural 
activities. The project logic model did not include multicultural elements, and these activities were 
rejeaed. After receiving additional requests for participation in multicultural activities, the 
stakeholders reviewed their program logic and found clear associations with multicultural issues 
and their desired changes in community health. With modification, the requests for participation 
were incorporated into the activities of the project. 

The specification of activities within the framework of program logic can provide an effertive 
process for managing project resources and maintaining the focus on desired system changes by 
systematically integrating new ideas into the rationale for the project. 

Outcomes— the short-term, immediate results of activities— are the third component of the logic 
model. Outcomes are specific, relatively immediate changes in the knowledge, state, and/or 
processes of system members and organizations. For example, consider the increased knowledge 
parents are likely to receive by participating in parenting classes as part of a family program for 
first-time youtlt offenders. Or consider the community change in providing increased affordable 
housing that allows shorter waiting time for families to receive housing assistance. In each of these 
cases, there are immediate changes — the outcomes — that are attributable to specific community 
activities. 

As with the specification of project activities, the specification of outcomes is an iterative 
process. Some outcomes will be identified from consideration of project activities. Other outcomes 
will arise from consideration of the health of the system and will include specific system strengths 
and address system needs. Still others will arise from consideration of desired changes in the 
fundamental nature and operations of the system — the system impact. 

Closely identified with outcomes is the desired systemic impact arising from the project. Impact 
is the long-term, substantial change in the health of the system. Impact is identified from 
consideration of the vision of system health that stakeholders share. Very often, impact is 
identified from a clear understanding of the present condition of the system and represents the 
vision of a healthy system. As we use it here, impact refers to the large, fundamental changes that 
will have long-lasting and far-reaching effects on the health of the system. These effects are the 
systemic changes that are described in the second article in this series (see the Western Center 
News, December 1994). 

As an example, one community prevention network seeks to have an overall impact of 
reducing children’s and youths’ illicit use of alcohol and drugs and reducing adult abuse of alcohol 
and drugs. Outcomes that will lead to these fundamental changes in the community include 
encouraging families to create and maintain positive guidelines for their children; fostering 
community attitudes that support no use of drugs and alcohol among children and youth and 
responsible use among adults; encouraging families that are socially isolated to be more connected 
to the community; and encouraging youth to develop healthy bonds to their family and 
community. As a set of expectations, outcomes and impact define the direction for the project, and 
aaivities that are rationally linked to the desired change in community health are matched to this 
direction. 

Program logic is the links among the health of the system, project activities, outcomes, and 
impact. It is built from shared craft knowledge and empirical evidence about effective systemic 
change efforts. Several lines of research provide a model of program logic that has proved helpful 
in community projea planning. Community-focused program logic can be organized in four 
components that describe two dimensions of internal and external qualities of healthy individuals 
and families. Aligned along the dimension of external qualities are community proteaive factors, 
described by Benard (199D, and stressful events, discussed by Holmes and Rahe (1967). These 
two com|X)nents describe the community support systems, organizations, and services that 
promote individual and family wellness, along with the life events that aeate stress for individuals, 
families, and communities. Very often this dimension, external from individuals and families, is 
where communities can develop immediate and responsive activities to improve the health of 



individuals, families, and the community. Developing responsive community support systems — 
developing organizations and services that promote individual and family wellness — can be 
decisive ways communities can help individuals and families deal with stressful life events. 

The internal dimension of health involves the risks individuals and families face and the 
resiliency they possess to withstand stress and change. Risk, described by Hawkins and Catalano 
(1992), refers to the traits and conditions that may limit the wellness of individuals and families. 
These traits and conditions reflea behaviors that limit, for example, social bonding, encourage 
dysfunctional behavior, and lead into strong alienation and rebelliousness. Resiliency, desaibed by 
Benard (1991) and Wolin and Wolin (1993), refers to the traits and conditions that mitigate risk 
and allow individuals to cope with stress. In general, resiliency includes social competence, 
problem-solving skills, and a sense of purpose and future. 

Communities can develop healthy individuals and families by encouraging the development of 
resiliency and addressing risk factors. Frequently, these dimensions are addressed in communities 
by focusing on increasing social bonding through community participation, developing consistent 
and high expectations for individuals and families, and encouraging individual and family growth 
and resiliency. 

The logic model provides a theoretical framework for developing and conducting the projea 
evaluation. From the model, the activities and outcomes can be evaluated following traditional 
evaluation strategies. Formative evaluation related to the implementation of activities, the barriers 
and solutions encountered in conducting the activities, and the direct outcomes of the activities 
can be documented. For example, a parenting course that is required for parents of first-time 
youth offenders can be monitored to determine the number of parents participating in each 
parenting class, the curriculum included in the classes, the parents’ attitudes toward the 
curriculum, the teaching methods used, and the knowledge gained by parents about their role in 
helping youth develop strong bonds with family and community. With a clear understanding of 
the activities and the rationally linked outcomes, the evaluation can be designed to assess the level 
of implementation and realization of desired outcomes. 

Project assessment requires the longitudinal evaluation of the change in system health. This 
focus on system condition and impact in the logic model, enclosing project activities and 
outcomes, can provide a systemic framework for understanding the substantial, long-term, and 
fundamental change in system health. Periodic profiling of the key indicators of system health can 
provide a comprehensive assessment of improvements in system health. With the rational linkage 
between this systemic framework and the activities and outcomes of the project, program 
effeaiveness can be assessed. 

In future articles in this series, specific evaluation techniques for monitoring activities and 
outcomes, as well as assessing systemic impact will be discussed. For now we have offered a tool 
for planning systemic projects that can help lead toward improvements in the health of the system. 

References 

Benard, B. (199D. Fostering resiliency in kids: protective factors in the family, school, and 
community. Pordand, Oregon: Northwest Regional Educational Laboratory 

Hawkins, J. and Catalano, R. (1992). Communities that care; action for drug abuse prevention. 
San Francisco, California: Jossey-Bass Publishers 

Holmes, T. and Rahe, R. (1967). Social readjustment rating scale. Oxford, United Kingdom: 
Pergamon Press 

Wolin, S. and Wolin, S. (1993). The resilient self. Westminister, Maryland: Random House 
Publishing 
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COMMUNITY DRUG PREVENTION NETWORK LOGIC MODEL 



Local Conditions 

Child Abuse 
Births to Teens 
Alcohol Outlets 
DtJls/MIPs 
Drug-afTected Births 
Media Advertisements 
Public Housing Needs 
Homelessness 
Early-leavers from School 
Suicide Attempts 
Discipline Referrals 
Juvenile Crime Rate 
Levels of AOD Use by 
Youth 

ParentA^outh Attitudes 
Public Policies 
Gang Afiiliation 
Bias Crimes 



Strategies 

A. Identify and promote family management programs that 
effectively address AOD risk faaors. (1,2,3) 

B. Strengthen and support clear community laws and norms 
regarding AOD use. (2,4) 

C. Improve consistency of enforcement of community laws 
and school policies regarding AOD use. (2) 

D. Educate children, youth, and parents about legal and social 
consequences of AOD use by children and youth. (1 ,2) 

E. Encourage reduced alcohol use at community events. (2,4) 

F. Promote increased AOD-free community events for children 
and youth. (2,4) 

G. Utilize media to promote messages about children and 
youth and reduce messages encouraging AOD use. (2,4) 

H. Identify and network with community agencies, families, 
and resources to address families’ social isolation, extreme 
poverty, and poor living conditions. (3) 

I. Woiic with schools and community to provide activities that 
promote bonding of youth and families. (2,4) 

J. Develop and maintain a partnership of key community lead- 
ers to address AOD risk factors. (1,2,3, 4) 

K. Educate youth and adults regarding cultural diversity and 
promote multicultural inclusion. (2,3,4) 



hnpact 

a. Reduce children’s and 
youth’s use of AOD. 

b. Reduce AOD abuse 
among adults. 



Notes: 1. AOD - Alcohol and Other Drugs 2 Parentheses include indexes of Local Conditions to Strategies to Outcomes to Impact. 



Outcomes 

1. Families will aeate and ite and 

maintain positive guidelines uidelines 

for their children, (a) a) 

2. Community altitudes ides 

about AOD use and abuse d abuse 

will improve. (a,b) 

3. Families who experience perience 

soda! isolation will be more 1 be more 

connected to the communi- ommuni- 

ty. (a,b) 

4. Opportunities for youth youth 

to CTeale healthy bonds with onds with 

school, family, congregation, igregation, 

and community will in- ill in- 

crease. (a,b) 
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Article 4, June 1995 



Developing Systemic 
Improvement Strategies 

By JOAN SHAUGHNESSY 

Northwest Regional Educational Laboratory 

The systemic improvement process is a paradigm shift to a different mode of problem solving. 
(Paradigm shifts are major changes in a prevailing set of assumptions and beliefs.) The motivation 
for making this shift is that a systemic viewpoint improves the program’s ability to address major 
issues in a coordinated, comprehensive, and meaningful way. 

Most organizational structures in our Western civilization are based upon a mechanistic 
viewpoint which suggests that each problem we address can be broken down into parts and 
subparts (Wheatley, 1992). Each environment — schools, health organizations, crime prevention 
agencies, and so on — ^works on its own part of the process in isolation from the others. Each 
operation in this complex process is done by a separate team of people using different skills and 
tools. The result is compartmi.'italization and specialization. Many different agencies, committees, 
or task forces are created, each one managing one portion of the work. While their approaches 
may be multifaceted, they usually lack a systemic focus. Subparts of a system can chug along on 
parallel tracks unaware of others doing similar work, or at times working counter to each other. 

In contrast, a systemic viewpoint maximizes agreement about what is most imp>ortant, facilitates 
coordination, and minimizes the chance tliat forces within the system are acting in contradiction to 
one another. 

To prevent this from being too abstract, let’s use a specific example. Suppose that alcohol and 
other drug (AOD) abuse is identified as major problem in your community. You know that this 
issue is being addressed by many parts of your system, each specializing in a specific set of 
solutions. Because these many gioups hold different and diverse perspectives, each may spawn a 
different way to address that problem. For example, one social service group may be trying to 
forestall the use of illegal substances by providing quality family time, whQe a school-based group 
may be trying to give teenagers more self-esteem by providing a “teens only” club. Both of these 
goals are admirable; both are inherently beneficial. ^S)^en these two subparts of the system each 
schedule a major event on the first Saturday in May, however, they will be acting at cross- 
purposes. Meanwhile, the more serious cause of the AOD abuse may be issues that are not dealt 
with by either group and no one in the system is attending to these underlying problems. This is a 
simplistic example, but it gives you a sense of what happens in a nonsystemic model. If there is 
no systemic perspective, there is no connection across the parts and there is no general agreement 
about what needs to be done to address problems. 

In designing systemic improvement, three important issues need to be addressed: what to do 
before you develop strategies for your system, what to consider during the selection of systemic 
strategies, and how to maximize success throughout the design and implementation phase. 

First Steps 

What are the essential elements needed before planning the intervention? There are many and, 
to be honest, they demand a great deal of commitment and energy at this first stage of the 
process. Systemic improvement is not a quick-and-easy fix. 

A first step in strategizing for systemic change is to define your system and identify who and 
what are included in the system (the stakeholders). You will need to ensure that stakeholders see 
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the value of collaborating with others in the total system. Gaining participation among people in 
different parts of the system is a vital step and it is best done by a cooperative rather than an 
individual effort. 

Secondly, creating a leadership team with members from various parts of the system will help 
focus the work beyond the narrow perspective of any one agency. Once formed, your leadership 
team needs to understand the existing interrelationships between the components in your system. 

Your third step will be to select one of several ways to foster this systemic viewpoint. Drawing 
a visual representation of your system or a diagram of the forces operating is one activity that 
helps stakeholders break out of their compartmentalized thinking. You will probably start with a 
simplistic sketch of the system and expand upon it as you learn more about the system. A matrbc 
or chan listing all the subcomponents and the major commitments in the system may also be 
helpful. 

Another way is to work on consensus about the desired systemwide outcomes. The logic model 
discussed in the article on Page 4 is an approach that prompts discussion of the factors operating 
in a system. Another help in this phase of the process is to use symbolism. For example, to 
broaden systemic thinking, we have shown aerial photos of river systems with all their many 
tributaries. We ask groups planning systemic change to contrast the river as a naturalistic system 
V 'ith the one they work within. Leadership teams begin to ask: “What binds a natural system like a 
river together?” They discover that the tributaries of the river connect for a singular, simple but 
powerful purpose — to follow the call of gravity. A river system works its way around many 
different types of obstacles to reach its goals. Leadership teams are encouraged to ask what kind 
of force would unite their system to achieve a common goal. The insights from such a discussion 
often can be profound and provocative. 

A fourth step to prepare for systemic thinking is to gather data about problems and issues 
facing die system. Systemic thinkers need to be aware of underlying causes, societal pressures, 
research-proven practices, and craft knowledge. To get to that point, your leadership team will 
need to systematically collect a variety of information. Articles about systemic improvement, books 
that guide data collection (such as those on Total Quality Management in schools), materials about 
school profiling and research syntheses of effeaive strategies are helpful documents at this stage 
of the process. Talking and listening to a variety of stakeholders throughout the system also 
provides valuable information. Consultants who can help you collect valid and reliable data or can 
recommend appropriate articles or information about research proven approaches are a good 
investment at this stage of the process. 

To continue with our example, just think of all the many unanswered questions you would 
have after you identified systemwide AOD problems. To formulate appropriate strategies you need 
to know which segment of the population is most susceptible to drinking problems. When do 
these problems occur? What programs are already in place in the community? Who do they serve 
and what kinds of successes do they have? What strategies really work and have data to prove if 
Will these strategies work in your community? Each of tliese questions can launch a series of 
individual research projects, but when you are thinking systemically, you’ll need to tie your 
investigations to one another. Your aim is to study the whole river, not just one of its minor 
tributaries. 

Another precondition before designing a systemic strategy is to maximize the communication 
and linkages among components of the system. Multiple communication processes will facilitate 
collaboration. Communication channels should be formal and informal, written and spoken, and 
should include the sharing of all data and information collected. In addition, communication 
networks must be two-way processes; the leadership team needs to share what they have learned 
and to listen to those in the many different parts of a system. In short, your fifth step is to ensure 
that multiple forms of outreach and input have been developed. 

The sbctli issue to face is securing widespread agreement about the direction you are heading 
and to develop goals that are truly systemic. Instead of focusing on narrow definition of issues, 
you will need to identify broad-based goals and to help stakeholders identify and articulate their 
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shared beliefs. (Issues related to selecting systemic goals were reviewed in the second article in 
this series.) 

When you have a bird’s-eye view of the problem, along with the facts about your system and 
agreement on the system’s goals, then you are equipped to plan your systemic improvement 
efforts. 



Designing Strategies 

Once you have decided what you want to accomplish, you will need to select implementation 
strategies that in fact work toward those goals, rather than those that are merely appealing or 
interesting to implement. 

Ideally, a holistic view needs to permeate all the work in the system. Shared beliefs that expand 
beyond the turf of one group need to be revisited periodically. Keeping those in the forefront will 
help focus the group to make choices along the way. All who will be part of tlie interventions 
need the opportunity to connect their work to other work underway in the system. 

As your leadership team designs action plans, they will need to ask three questions to verify 
that their strategies retain a systemic focus. First, do all stakeholders have a chance to participate 
appropriately in the design and implementation of the plan? Secondly, are the action plans 
integrated’ Thirdly, does the plan provide services needed to all who share in the problem? 
Planners need to be sensitive to the needs and motivations of diverse cultural groups so they can 
incorporate strategies that are appropriate for all. 

Stakeholder Participation 

Your leadership team may have a strong sense of the systemic problems, but it cannot carry an 
effort forward without stakeholder participation. Approaches that bring large groups together for 
several meetings, such as the Future Search conference described by Weisbrod (1992), help a 
program to select a systemic goal and design specific action plans to address a common problem. 
Those who will be providing the services should also participate in these events. Be prepared to 
bring stakeholders together several times throughout the process to maintain their interest and 
maximize their input. 

When engaging diverse stakeholders, structure time together to ensure that collaborative 
participation is a positive experience. Build in personal sharing and team-buUding activities and 
encourage social interaction. Establish an atmosphere of trust and openness, and be prepared to 
diffuse confrontation at meetings. Revisiting shared beliefs can help this process. 

Ensuring that the efforts are integrated demands formation of partnerships across agencies. 
Partnerships work best when all participating groups benefit from the collaboration. Often one 
group, such as the leadership team, serves as the communication link and/or manager of the 
action plan, but the group continues to form additional groups to oversee portions of the process. 
It is helpful if each new group revisits the concepts of systemic thinking and their commitment to 
the systemic goals before and during their work. 

To be comprehensive, the effort needs to reach all who are cormected to the problem. 
Intervention strategies should avoid being focused on just a small segment of the population (for 
example, those who self-select to participate.) For the intervention strategy to be systemic, the 
service providers need to identify the underlying cause and then reach out to affected populations 
and expand services broadly throughout the system. In short, the combinations of solutions 
generated should work with all segments of the population. 

Maximizing Success 

If you are going to invest this amount of effort, you will want to ensure that the systemic work 
you are doing is successful. Here are some recommendations that can help maximize success 
throughout the effort: 

• Make sure there are many different and diverse groups participating in the planning and the 
action steps. It is particularly helpful to include representatives of populations being served by 
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the program in the planning process. Representatives also need to be accessible to all 
stakeholders. 

• Keep revisiting the mutual goals and shared values that you have developed. These will help 
you overcome resistance to systemic work. When resistance occurs, acknowledge it and work 
through it constructively. 

• Make sure the rewards throughout the system support collaboration. Those who are being 
discouraged from doing collaborative work will not continue with the process. 

• Continue to introduce outside learning from time to time to spur thought. Encourage 
continuous reflection and thinking about the process. 

• Realize that the strategies you have developed are work “in progress." You will keep learning 
and developing newer and belter ideas as you go along, so encourage flexibility. You will 
want the plan you develop to be adaptable and at times even opportunistic. When new data 
or information becomes available or new strategies are generated, add these to your plan. 

• The reverse is also true. Strategies for systemic improvement that are not effective need to be 
abandoned. The ability to make midstream corrections can streamline the effort and make it 
more cost effective. 

• To add or drop pans of your plan, evaluation of the process is essential. 
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How to Design a Systemic 
Evaluation Process 

By ROY KRUGER and PATRICK WEASF L HEAD 

Northwest Regional Educational Laboratory 

The previous articles in this series identified four key elements we will build upon in 
developing strategies for a systematic approach to systemic evaluation of community improvement 
programs: 

• Systems are dynamic and hierarchical in nature (Article 1) 

• The “logic model” provides a systematic tool for systemic monitoring and assessment (Article 3) 

• A systemic perspective is an important component of a systematic monitoring and impact 
evaluation process (Article 4) 

• Systemic program goals are comprehensive and complex (Article 2) 

Systemic evaluation is a process of assessing the impact of activities that are designed to change 
complex behavior and norms. It is distinguished from older, more linear evaluation models in that 
it takes a broader, more comprehensive perspective on programs and their effects. This new 
systemic perspective acknowledges the extraordinary complexity of human systems. Rather than 
focusing narrowly on isolated pieces of a system, then, systemic evaluation looks at a set of 
interrelating variables within a particular context (Yap and Arrasmith, 1993). Acceptable research 
evaluation practices require that the evaluator know what the program is intended to accomplish; 
who or what is supposed to benefit from the intended change; who operates the program; and 
what activities or strategies are employed to create change (King et al, 1987). In collecting data 
(either through quantitative or qualitative research methods), the evaluation process should be 
designed to adequately reflect the expectations of diverse stakeholders while clearly presenting 
the constraints on the program. The basic purpose of the evaluation also must be clear. 

Recent school reform initiatives, such as the national Goals 2000 legislation, have spelled out 
the expectations of policymakers for program accountability. In calling for systemic reform, such 
initiatives also call for systemic evaluation. As indicated in The State’s Role in Effecting Systemic 
Change: A Northwest Depiction (Hagans et al, 1992), systemic evaluation includes multiple targets 
that are malleable and have wider latitudes than those in nonsystemic evaluation. How can one 
collect empirical data and not let old standards get in the way of the systemic reform movement, 
especially in the systemic evaluation process? To answer this question, one should understand the 
dynamics and hierarchical nature of systems. 

Nature of Systems 

The systems model has become an inaeasingly used comprehensive tool for illustrating and 
understanding complex interrelationships that exist between individuals, organizations, and 
communities. The systems model provides important insights for designing a systematic evaluation 
process to monitor the progress and impact of community improvement programs. 

Systems are dynamic entities, comprised of subunits called elements. The elements of a 
particular system interrelate in a unique way, producing distinctive attributes that differentiate that 
system from all other systems. Systems analysts describe this phenomenon as the whole (system) 
t^ing greater than the sum of the parts (elements). 

For example, one system we all come in contact with each day is water. If asked to list the 
attributes of water, most people will include “wetness” somewhere on their list. It is the unique 
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interaction of the elements hydrogen and oxygen that produces the attribute “wemess.” However^ 
further investigation reveals that neither hydrogen nor oxygen alone exhibits the quality “wetness.” 
For this reason, it is crucial for the evaluator to develop an assessment process that examines both 
the interactions of the individual elements and the system as a whole. 

Each community is a system that exhibits its own unique character. Although Coos Bay and 
Newport, for example, are both small communities on the Oregon Coast, each exhibits a unique 
essence. Newport, for example, benefits economically from its proximity to Portland and enjoys a 
thriving tourist industry, while C .os Bay, a port town, is struggling economically in the face of the 
timber slow-down in the Nortfiwest. These economic differences are among the factors affecting 
the interrelationships of the schools, businesses, civic groups, and ocher community elements in 
each town. Problems present in Coos Bay may or may not be present in Newport, and 
improvement programs appropriate for Newport may or may not be appropriate for Coos Bay. 

Each system is part of a “continuum of systemness,” wherein each system is an element in the 
next higher level system, and each element of a system is its own system with subelements at the 
next lower level. An example is the school system. As a system, it is comprised of students, 
teachers, and administrators. The school is also an element of the community. To understand the 
impact of an improvement program designed for the school necessitates understanding the 
program’s effects on each element within the school system and on tlie school system as a total 
entity. 

The Logic Model 

The “logic model” described in the third article of this series (June 1995) provides an effective 
tool for systematically analyzing the separate activities of an improvement plan within the 
framework of a systemic, communitywide approach for improving local conditions. The logic 
model assists the analyst in assessing how multiple, often hierarchical activities relate to each 
other, how they relate as a whole to the surrounding community conditions, and whether they are 
likely to bring about systemic change within the community. The chart on page 20 is a suggested 
prototype for a community action plan systemic logic model. 

One of the initial steps in developing a systemic evaluation process is assessing whether and to 
what degree program objectives are systemic. The second article in this series (December 1994) 
suggests that program objectives are systemic if they are; 

• Sustainable: Held to be important by the community without special funding 

• Potent: Understood, valued, and embraced by a broad range of community stakeholders 

• Infusive: Built upon the community’s existing mission, resources, and infrastructure 

• Pervasive: Multifaceted and inclusive in their desired outcomes 

• Coherent: Create and encourage congruence among all levels of the system continuum 

Focus groups, surveys, and personal interviews of key community leaders are commonly used 
methods for ascertaining the extent to which goals meet these five systemic criteria. The 
evaluator’s major goal through this process is assessing the extent to which program outcomes are 
consistent with the overall long-term expectations and desires of the community. The closer the fit 
between community desires and program outcomes, the more likely it is that the program will 
achieve its goals. 

Other Influences 

Besides meeting the above criteria, an effective evaluation process must further assist the 
analyst in differentiating between the effects of three types of influences on local conditions: 
random influences, external influences, and program influences. 

Random influences; Random influences can momentarily affect local conditions, but in faa 
do not have a systemic or long-term influence on those conditions. An example might be the 
temporary effect of unseasonably hot weather on alcohol consumption. 

External influences; External influences that are outside the community’s control also affect 
local conditions. An example of such a factor is the influence of the state economy upon alcohol 



consumption. Ruhum (1995) found in a study of alcohol consumption patterns between 1975 and 
1988 that as the economy expands, alcohol consumption increases and visa versa. A $1,000 
increase in state per capita income increased expected liquor consumption by 1.2 percent, and a 
$1,0(X) decrease in state per capita income decreased experted consumption by 1.5 percent. A 
1 percent increase in the employment rate increased the expeaed consumption of liquor by 
1 percent. To accurately monitor the impact of specific alcohol consumption improvement 
programs on local conditions, a systemic evaluation process must be sensitive to the influences of 
these factors that are extraneous to the system described by the logic model. 

Program influences: If we consider our community improvement program as a system to 
influence the community, then our evaluation must be sensitive to the life cycle of the program to 
accurately assess its impact. Every program evolves through five stages of community influence 
during its lifetime: introduction, moderate influence, rapid influence, saturation influence, and 
declining influence. 

When a new program is introduced into the community (introduction stage) it has very litde 
influence beyond the initial program planners. Few community people know of or have 
participated in the program. The program planners will need to spend time and other resources to 
promote the program and involve community stakeholders in the program. 

As members of the community begin to learn about and participate in the program, its impact 
grows beyond the initial planners (moderate influence stage). During this stage, the planners are 
still working diligently to promote the program and increase participation across the community. 
As more and more people become involved in the program, the impact of the program enters the 
rapid influence stage when the momentum of activities and community involvement increases its 
influence upon the community. 

During the rapid influence stage, planners spend less time on promoting the merits of the 
program and more time maintaining momentum and providing opportunities for participation. 
Eventually, a program will reach the saturation influence stage in which few potential new 
participants remain. 

It is during the saturation influence stage that evaluators should conduct a summative 
assessment of the program’s impact. The evaluation should determine whether the program 
brought about change that is systemic and therefore sustainable or change that is peripheral and 
therefore fleeting. If a program has not brought about systemic change in local conditions by the 
time it has reached this stage, it is likely to enter the declining influence stage. At this point, the 
program’s costs outweigh its benefits. When this occurs, planners may want to consider replacing 
or modifying Qie program, once again propelling it into an introductory stage. 

If we are to utilize evaluation reports for measuring systemic movements, then it is difficult to 
manage goals that are not clearly understood by and acceptable to everyone. The question is, do 
we simply “fix the parts” or do we cast our net more widely to encompass the system as a whole? 
Margaret Jorgensen (1993) indicates that to support systemic reform, we need to assess the 
performance of complex, holistic tasks and not simply snippets of these performances. How, then, 
does one implement systemic evaluation with goals that are sustainable, potent, infusive, 
pervasive, and coherent* 

Developing a Process 

One of the first steps in developing a systemic evaluation process is to list some of the key 
elements or charaaeristics of your organization or community. "What makes your organization or 
community special? How is it unique? Who are the individuals involved and how do they 
interrelate? In answering these questions, we need to evaluate all aspects of the systemic 
movement, develop an assessment instrument tliat measures the previous goals, and provide an 
analysis that is coherent and addresses accountability in meeting intended goals. 

A second step is to list the major goals and core values of your organization. What is your 
organization trying to accomplish? What are your organization’s basic underlying or core values? 
Out of this process should emerge your objectives, strategies, and activities, which should be 
specific, measurable, and designed to be accomplished in a definite time frame. There should be a 
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logical relationship between aaivities and the desired long-term systemic changes. If you want to 
influence community behavior through social bonding of individuals, for example, your program 
aaivities should logically encourage social bonding. 

A third step is to invite community comment regarding the congruence of your goals, 
objectives, strategies, and activities through focus groups, surveys, and/or interviews. The results 
of your study may suggest the adoption of other strategies to influence community behavior or 
norms. 

A fourth step is to develop an evaluation that will assist in analyzing the degree of influence the 
program has had on community behavior and norms. This model should allow for analysis of 
both short-term, intermediate outcomes and long-term, systemic outcomes. The intermediate 
outcomes relate to the success of implementing various activities, while long-term outcomes relate 
to community impact. 

Analysis and Reporting 

When preparing an analysis, the evaluator must possess two very important attributes; 
knowledge about and understanding of the systen- being evaluated, and a caring attitude about 
the results. 

Because the influences on changes in community behavior and norms are complex, the 
evaluator must avoid assuming a causal relationship between community program interventions 
and benchmark statistics. For example, it is possible that community intervention is having a 
systemic impact, while at tlie same time other influences are causing benchmarks to move in 
opposition to the desired direction (for example, the use of drugs may go up even though the 
intervention is having a positive impact on members of the community). Because systems have 
many variables that are not controlled, any one of them might have a causal effea on the desired 
change. 

When the best research findings and the best evaluation models are used to design and assess a 
community improvement program, we can logically conclude that our interventions have had 
some causal effea on the system at some level. With this in mind, evaluators may want to be 
involved in the planning stages, assisting community planners to develop a comprehensive and 
focused interrelated set of interventions grounded in research and linked to desired outcomes. 
This would deter communities from assuming that one activity, such as a “red-ribbon” parade, will 
change community behavior or norms. The logic model then becomes a strategic tool for 
accomplishing this task. If interventions are based upon what research says works in changing 
behavior and norms, we can assume that these activities will have a systemic influence on the 
community. 

The analysis also can assess the effectiveness of your program in achieving intermediate goals. 
An intermediate goal for a school, for example, might be to establish 25 mentoring partnerships — 
just one of a comprehensive set of interventions designed ultimately to reduce tobacco, alcohol, 
and other drug use among youth. If the mentorships were in fact established, we could say the 
school had achieved an intermediate goal, and we could also say that these intermediate outcomes 
are influencing our long-term systemic goals. We would incorporate this process for all of the 
activities or interventions we adopted. 

The important point is that we can make a logical connection between intervention and impact, 
while we couldn’t make that same connection between intervention and some specific benchmark. 
If we adopted this process we could also use our yearly results as a planning tool. We could 
assess whether the program had a systemic impact on the lives of those individuals directly 
involved in the intervention (that is, we could test whether it works or not). We could assess 
whether our goal of so many mentorships was attainable (that is, whether we need to devote 
more or fewer resources). Ultimately, we could assess whether a given intervention should be 
eliminated or others should be adopted to strengthen our program. 

If additional planning or program modification is needed, then a face-to-face meeting between 
systems management and systems users to discuss findings is imperative. Discussions involving 
recommendations for continuation, modification, or rejeaion of various program components are 
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best conducted in person. Such discussions would focus on whether efforts were sustainable^ 
potent, infusive, pervasive, and coherent. 
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Community Benchmarks 

1. Pregnancy rate per 1,000 
females ages 10-17. 

2. Percentage of children 
kindergarten teachers believe are 
prepared to partidpaie 
successfully in school. 

3. Percentage of drug-free babies. 

4. Percentage of drug-free teens. 

5. Percentage of safe child care 
facilities meeting basic standards. 

6. Percentage of high school 
students enrolled in 
vocational/educational piograms. 

7. Overall crime rate. 

8. Percentage of work force with 
strong adaptability skills. 



Conditions 

1. Low citizen trust of 
government effecting their 
involvement and 
commitment 

2. Inadequacy of the 
delivery of needed family 
services. 

3. Lack of citizen 
involvement in solving local 
community issues. 

4. Lack of support 
community and government 
support to gain a sense of 
health and family wellness. 



Activities Outcomes Impact 



1. Establishing Community 
Human Services Teams: 

A. Skill-building and tool 
development 

B. Discussing problems and 
issues. 

2. Developing clear and 
shared vision of health and 
wellness for families and 
community. 

3. Identifying and involving 
stakeholders and key partners. 

4. Engaging stakeholders in 
finding solutions to problems. 

5. Facilitating alignment 
between public and private 
agencies and service providers 

6. Linking of community and 
county together in a systemic 
alliance. 



1. Establishment and 
strengthening of community 
progress teams who possess 
new problem solving skills 
and tools. 

2. Clear and shared vision of 
family and community 
health. 

3. Communitywide commit- 
ment and participation m 
strategy development and 
implementation. 



1. Develop healthy families, 
children, and other commu- 
nity groups, 

2. Develop a community 
and team shared vision of 
coordinated services that 
encourage healthy families, 
children, and community 
groups. 
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Using Evaluation Findings in 
Improvement Efforts 

ByCHANGHUAWANG 

Northwest Regional Educational Laboratory 

Before we get into the discussion on the use of evaluation information to strengthen strategies 
for systemic improvement, it will be helpful to review the concepts of “systemic improvement 
and “systemic goals.” As explained in the first article of this series, die improvement is systemic 
when it: 

• Is accepted within the components of the existing system 

• Spreads through the entire system 

• Has power to make visible, desired differences, both in terms of process and outcomes 

• Strengthens linkages among components of the existing system 

• Becomes part of and improves the culture of the system 

Following the same line of thinking, Steve Nelson in the second article of this series proposed a 
five-point test for a systemic goal. He stated that the goals of a program are more likely to be 
systemic if they are; 

Sustainable. Would the organization continue to hold this goal in the absence of special 
funding or programs? 

Potent. Is this goal understood, valued, and embraced by a broad range of stakeholders in the 
community? 

Iniuslve. Does this goal build upon the existing mission, resources, and infrastructure of the 
organization’ 

Pervasive. Is the goal multifaceted and inclusive in its desired outcomes? 

Coherent. Does the goal create and encourage congruence among the levels and dimensions 
of our system? 

A systemic evaluation as presented in previous articles should be designed to provide 
information in all of the above areas. Three key concepts— system, linkages, and stakeholders— 
can be discerned from the five criteria for systemic improvement and the five points for systemic 
goals. Expanding on these key concepts, then, we find that information gleaned through a 
systemic evaluation can strengthen strategies for systemic improvement by; 

• Defining and illuminating the system for improvement 

• Strengthening linkages among components of the existing system 

• Involving major stakeholders within the system 

Defining the System 

Nonsystemic evaluation tends to focus on isolated outcomes such as test scores without looking 
into the intricacies of relationships within the system. The results of nonsystemic evaluation often 
do not reflect what is really happening and are of little use for program improvement. For 
example, the Title I/Chapter 1 evaluation process was very systematic, almost to the point of being 
mechanical, but it was not systemic. Norm-referenced test scores were used to measure student 
achievement at the school, district, state, and national levels. The process was designed to 
produce a common measure on student achievement to be reported to Congress, not to provide 
information that would help program staff understand the relationships between program 
implementation and student outcomes. Consequently, the results of Title I have been of little use 



for program improvement at the local level. Although Congress has encouraged local programs to 
do more than what was required for evaluation, limited resources and misunderstanding of 
evaluation have meant that Title I administrators rarely have gone beyond compliance. 

Nonsystemic evaluation provides us with fragmented information and divides a whole 
experience into separate, isolated parts. In contrast, systemic evaluation starts with a careful study 
of the major relationships within the system. Great efforts are made to determine causal 
relationships that lead to certain outcomes. The goal of the evaluation is to find out how a 
program can manipulate ceitain variables (use various interrelated strategies) within the system to 
achieve desired outcomes. Information collected through systemic evaluation will be used to 
check whether goals set for a program are systemic by using the five-point test mentioned above. 

Before a program can set out to achieve its desired outcome, the system or context in which 
the program is operating needs to be defined. In the third article in this series. Dean Arrasmith 
describes the “logic model,” whose first component is defining the context or the condition of the 
system. The other compxjnents of the model are: the interventions or activities to be conducted, 
the expeaed outcomes related to the interventions, and the impact of intervention on the system. 
The logic model is often desaibed as a series of steps in a sequential order where one step builds 
on another. 

At the beginning of a program, the issues and problems that the program is supposed to 
address are defined, usually through a needs assessment or interviews with key stakeholders. For 
example, in the case of community drug prevention, these issues and problems may include, but 
are not limited to, child abuse, births to teens, alcohol outlets, drug-affeaed births, public housing 
needs, suicide attempts, and juvenile crime rate. These problems and issues clearly indicate the 
scope of the program and relative parameters of the system in which these issues and problems 
are dealt with. 

The evaluation can provide further information regarding existent resources available within the 
system. Based on this information, the program can identify strategies for interventions or activities 
for drug prevention within the community and define expected outcomes and impact of these 
strategies on the program. A systemic evaluation should be an integral part of a program designed 
for systemic improvement. Evaluation is not the end but a mechanism to monitor and stimulate 
change. 

Involving Stakeholders 

When the system is defined and key issues and problems are identified, the next question for 
the evaluation is: Who are the stakeholders of the program? Or, phrasing the question differently, 
who is going to use evaluation information and who can make a difference in the system? 
Research on knowledge utilization suggests that the successful application of knowledge requires 
two simultaneous conditions. First, the information must have a visible bearing on a perceived 
problem. Second, there must be a continuous and consistent dialogue between those who gather 
and provide information and those who must use it. 

The first condition requires consensus on the questions to be asked in the evaluation. To ask 
the right questions is almost a prerequisite for any use of evaluation information. Evaluators need 
to understand the system and their collaboration with key players within that system to reach an 
agreement on what is important to ask in the evaluation. This is often recorded in the evaluation 
design. 

The second condition suggests that stakeholders need to be involved throughout the evaluation. 
It is not uncommon to find presence of all stakeholders at the initial stage of the evaluation, but as 
the evaluation progresses, the program director is perhaps the only person who still knows the 
status of the evaluation. Therefore, it is important to keep all stakeholders informed of the 
progress of the evaluation and obtain their feedback on all major issues relating to the evaluation 
from beginning to end. In this sense, a systemic evaluation is a continuous and consistent dialogue 
between stakeholders and evaluators. Stakeholders need to be involved in interpreting the 
evaluation data that are provided by evaluators. When this occurs, the evaluation information is 
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more likely to be used effectively to monitor or verify whether target groups have been served as 
the program intended. 

In some cases, the source of funding for the evaluation may be an indication of who cares 
about the evaluation results. For some federally funded evaluation projects, evaluation questions 
often are already defined by the funding agency and are not particularly tailored for local interest. 
Asking a local agency or agencies to match certain evaluation funds and include evaluation 
questions of local interest may be an important step to obtaining active participation of local 
stakeholders and shared ownership of the evaluation results. 

Strengthening Linkages 

One of the criteria for systemic improvement is that it is likely to strengthen the linkages among 
components of the existing system. The logic model can be used as a tool to monitor such 
linkages. The beauty of the logic model is the simplicity of the sequence of steps as described in 
Article 3 as “rational, empirical, and intuitive.” 

However, this model could also be misleading. The connection of the four basic components of 
the model could be misconceived as simple and sequential: one component leading to another. 
Systemic evaluation, however, involves multiple levels of relationship that may not be sequential 
as described in the logic model. Therefore this model has to be used with caution. 

Within a system where relationships between different components are “fluid and dynamic,” 
there is no simple one-to-one causal relationship. One single outcome may result from multiple 
factors existing in the system. Systemic evaluation seeks to trace and explain such complex 
relationships by using multiple variables and qualitative study of the context surrounding such 
relationships. Systemic evaluation may not be able to pin down one or two solutions to certain 
problems. But evaluation information obtained through a systemic evaluation may help reduce the 
uncertainty of decisionmakers in seeking solutions to problems, while fostering more meaningful 
relationships between and among various components within the system. 

Systemic evaluation should be viewed as an integral part of a program. It is a built-in 
mechanism for seeking the information that will help strengthen strategies for systemic 
improvement. Because of its complexity, systemic evaluation may require redefining the roles of 
an evaluator. In addition to being an external, objective observer, the evaluator might also be a 
participant in a program. The appropriate relationship between evaluators and their clients may 
not be as distinct as traditionally defined. 

For more information on systemic evaluation for prevention programs, contact Jerry Adams at 
( 503 ) 275 - 9484 . 
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